Merge v4.0.0 into main - #416

…eline

[KDM-CLI-FEAT-339] feat(core): Phase 0 - Baseline pipeline mapping and contract freeze

Implements a decorator-based ToolRegistry with Ollama-compatible JSON schema
generation and strict least-privilege enforcement for Kubernetes and Docker
inspection tools.

## Changes

### agents/tools/__init__.py
- New package exposing ToolRegistry and the tool_registry singleton.

### agents/tools/registry.py
- ToolRegistry class with @register decorator.
- Auto-generates Ollama function schemas via inspect.signature.
- Whitelist enforcement: raises ValueError for unregistered tool calls.
- 10-second asyncio timeout per execution; returns error dict on expiry.
- Handles both Python type objects and string annotations (Python 3.13+).

### agents/tools/kubernetes.py
- get_pod_status(namespace, pod) -> phase + containerStatuses.
- get_container_logs(namespace, pod, container, tail_lines, previous) -> bounded log lines (max 500).
- get_pod_events(namespace, pod) -> Warning events sorted by lastTimestamp.
- get_deployment_spec(namespace, deployment) -> replicas, selector, template.
- RBAC/Forbidden errors surface as structured dicts.
- All responses scrubbed by core.sanitizer.redact_sensitive_data.

### agents/tools/docker.py
- get_docker_container_inspect(container_id) -> State, ExitCode, OOMKilled, RestartCount.
- get_docker_logs(container_id, tail) -> bounded stdout/stderr lines (max 500).
- All responses scrubbed by core.sanitizer.redact_sensitive_data.

### agents/tests/test_tools.py
- 20 unit tests, all passing (pytest + pytest-asyncio).
- Covers: whitelist enforcement, schema generation, type mapping, tail
  clamping, --previous flag, Warning event filtering/sorting, Docker state
  extraction, timeout behaviour, and a final check that no mutating tool
  names are registered.

Closes KDM-cli#273

…safe-typed-tool-layer

feat(agents): Phase 2 - Safe Typed Tool Layer & Least-Privilege Registry

Implements the normalized, immutable evidence layer as the single source
of truth for all specialist agents and deterministic rules.

Core Architectural Rule: KDM collects facts; agents reason over facts.

Changes:
- agents/core/__init__.py: Export core models
- agents/core/evidence.py: CollectionStatus enum, Target, EvidenceItem,
  and EvidenceBundle dataclasses with to_dict/from_dict round-trips
- agents/core/context.py: AnalysisMetadata and AnalysisContext wrappers
  with UUID-based run identification
- agents/core/sanitizer.py: Recursive credential redaction utility using
  case-insensitive regex for password/secret/key/token/auth/credential/cert
- agents/tests/test_evidence.py: 42 unit tests covering bundle creation,
  status mapping, JSON serialization round-trips, deeply nested secret
  masking, PERMISSION_DENIED and TIMEOUT status handling

Closes KDM-cli#272

- evidence.py: remove EvidenceItem.create (6-arg biomarker) and use
  field default_factory for UTC timestamp generation; require collected_at
  in from_dict
- context.py: add __post_init__ invariant rejecting target mismatch with
  evidence.target across all construction pathways; remove unused field import
- sanitizer.py: use explicit stack traversal instead of mutual recursion to
  prevent RecursionError; redact values of Kubernetes env entries whose
  sibling name matches sensitive patterns
- __init__.py: sort exports in __all__ alphabetically to satisfy RUF022
- conftest.py: add pytest configuration ensuring agents/ is in sys.path
- test_evidence.py: update callers to direct EvidenceItem instantiation,
  add target mismatch invariant tests, missing collected_at KeyError test,
  deep recursion sanitizer test, sibling env var secret test, and full
  docstring coverage across all test methods

…canonical-evidence-model

feat(agents): Phase 1 - Canonical Evidence Model and Collectors

…into cohesive modules

- sanitizer.py: extract _process_dict, _process_list, and _prepare_child
  helpers to eliminate nested complexity and satisfy CodeScene code health rules
- split monolithic test suite into highly cohesive test_evidence.py,
  test_context.py, and test_sanitizer.py matching agents/core modules

… duplication

…odescene-and-cohesion

refactor(agents): resolve CodeScene code health rules and modularize tests for Phase 1

Implements an in-memory auditable Evidence Store and Tool Audit Trail that
records every tool call with millisecond timings and links diagnostic findings
to supporting evidence IDs and tool call records.

Changes:
- agents/core/store.py:
  - ToolCallRecord: captures call_id (call-<hex8>), agent_role, tool_name,
    arguments (with sensitive key scrubbing), result, error, started_at,
    duration_ms, and status (pending/success/error/timed_out).
  - FindingRecord: captures finding_id, agent_role, claim, confidence,
    and evidence_refs with to_dict/from_dict round-tripping.
  - EvidenceStore: thread-safe in-memory store supporting record_tool_call,
    record_finding (validates non-empty evidence_refs), get_provenance
    (resolves finding to supporting tool calls), export_audit_log, and JSON
    serialisation.
- agents/core/__init__.py: Export EvidenceStore, FindingRecord, and ToolCallRecord
  in alphabetical order in __all__.
- agents/tools/registry.py: Instrument ToolRegistry.execute with optional agent_role,
  automatic duration measurement, error/timeout handling, argument redaction,
  and automatic recording to EvidenceStore.
- agents/tests/test_store.py: 24 unit tests covering ToolCallRecord defaults/scrubbing,
  FindingRecord validation, EvidenceStore recording/provenance/audit export,
  thread safety, and ToolRegistry execution integration.

Closes KDM-cli#274

…tool-audit-logging-and-evidence-store

feat(agents): Phase 3 - Tool Audit Logging and Evidence Store

Implements a deterministic diagnostic rule engine that diagnoses canonical
Kubernetes failure signatures in < 5ms before invoking any LLM turns.

Part of Milestone v4.0.0 — Phase 4 of the Multi-Agent SRE Architecture.

Changes:
- agents/rules/base.py:
  - BaseRule abstract class with evaluate(EvidenceBundle) -> Optional[RuleMatch]
  - RuleMatch dataclass with rule_id, title, root_cause, confidence, evidence_ids,
    confidence range validation, and to_dict/from_dict round-trips
- agents/rules/oom.py:
  - OOMKilledRule detecting exit code 137 or reason OOMKilled from container status
- agents/rules/image_pull.py:
  - ImagePullRule detecting ErrImagePull and ImagePullBackOff waiting states
- agents/rules/crashloop.py:
  - CrashLoopRule detecting CrashLoopBackOff waiting states with exit code > 0
- agents/rules/probes.py:
  - ProbeFailureRule detecting liveness/readiness/startup probe failures from events
- agents/rules/scheduling.py:
  - SchedulingRule detecting FailedScheduling events with Pending pod phase
- agents/rules/engine.py:
  - RuleEngine dispatcher with evaluate_all, get_primary_match, and should_bypass_llm,
    sorting matches by confidence descending with sub-millisecond execution
- agents/rules/__init__.py:
  - Alphabetically sorted package exports
- agents/tests/test_rules.py:
  - 40 unit tests covering all positive matches, negative cases, edge cases,
    and execution benchmarking (< 10ms threshold)

Closes KDM-cli#275

…de health rules

- engine.py: add logger.exception with rule_id when rule execution fails
- oom.py: assign 0.8 confidence when exit code 137 is present without OOMKilled reason;
  prevent sidecar misattribution by restricting fallback scans when target container is present
- image_pull.py: prevent sidecar misattribution by restricting fallback scans when target container is present
- crashloop.py: prevent sidecar misattribution by restricting fallback scans when target container is present
- scheduling.py: restrict pod phase lookup strictly to ev.pod.status and ev.pod.phase
- reduce method cyclomatic complexity, nested depth, and bumpy road conditions across all rules
- test_rules.py: parameterize tests to eliminate code duplication, improve cohesion,
  and reduce file length from 700 to 489 lines (< 600 line threshold)

…ive exit code for crashloop

- crashloop.py: require parsed exitCode > 0 (rejecting missing/unparseable/negative exit codes) per issue KDM-cli#275 check; flatten methods to reduce mean complexity to <= 4.0
- oom.py: decompose _detect_oom into _detect_for_target and _scan_containers to eliminate Bumpy Road and reduce mean complexity to <= 4.0
- image_pull.py: decompose _detect_image_pull_failure into _detect_for_target and _scan_containers to eliminate Bumpy Road and reduce mean complexity to <= 4.0
- scheduling.py: split _is_scheduling_event and _inspect_pod_phase into modular helper functions with 0 complex conditionals and complexity <= 4.0
- test_rules.py: add test scenarios for missing, unparseable, and negative exit codes in CrashLoopRule

… comments

…deterministic-rule-engine

feat(agents): Phase 4 - Deterministic Diagnostic Rule Engine

…, Resource)

Implements structured specialist agents (Runtime, Config, Resource) using
Ollama with strict JSON schemas to eliminate hallucinations and adhere to
a canonical SpecialistReport contract.

Part of Milestone v4.0.0 — Phase 5 of the Multi-Agent SRE Architecture.

Changes:
- agents/specialists/base.py:
  - BaseSpecialistAgent abstract base class taking client: ollama.Client and model: str
  - run_investigation(bundle: EvidenceBundle) -> Dict[str, Any] passing format="json"
    and options={"temperature": 0.1}
  - Safe JSON extraction, markdown code fence stripping, and fallback report recovery
    on JSONDecodeError, KeyError, or execution timeouts
  - Backward-compatible analyze() method supporting legacy council workflow
- agents/specialists/runtime.py:
  - RuntimeLogAgent: Senior Linux & Container Runtime Diagnostics Specialist
  - Extracts container exit codes, termination signals, and previous/current logs
  - Explicitly notes empty previous logs rather than hallucinating application errors
- agents/specialists/config.py:
  - ConfigDependencyAgent: Kubernetes Declarative Configuration Specialist
  - Inspects missing volumes, missing ConfigMap/Secret keys, waiting states, and probe timeouts
- agents/specialists/resource.py:
  - ClusterResourceAgent: Cluster Capacity & Linux Cgroups Specialist
  - Inspects node pressure conditions (MemoryPressure, DiskPressure), QoS classes
    (Guaranteed, Burstable, BestEffort), and cgroup OOM terminations
- agents/specialists/__init__.py:
  - Package exports and legacy SynthesizerAgent compatibility
- agents/tests/test_specialists.py:
  - 20 unit tests with mocked Ollama chat responses covering valid JSON, markdown fences,
    malformed text recovery, Ollama timeouts/exceptions, domain prompt extractions,
    and schema validation across all 3 specialist agents

Closes KDM-cli#276

… comments

- agents/specialists/base.py: pass parsed evidence through _normalize_report in analyze();
  eliminate compound conditionals in markdown fence stripping
- agents/specialists/runtime.py: eliminate all complex conditionals by using single-clause
  guards and helper predicates; flatten status matching
- agents/specialists/config.py: extract envFrom and env.valueFrom ConfigMap/Secret references
  across regular and init containers; eliminate complex conditionals
- agents/specialists/resource.py: ensure QoS Guaranteed match requires both CPU and memory
  equality; eliminate complex conditionals in QoS calculation and node condition formatting
- agents/specialists/__init__.py: log exception on legacy loader failure; split compound check
- agents/tests/test_specialists.py: add tests for analyze() evidence normalization,
  envFrom/valueFrom manifest extraction, and CPU-only QoS Burstable classification

…cialists

- base.py: eliminate Bumpy Road Ahead in _extract_content, _strip_markdown_fences, and _normalize_report by extracting flat helpers
- runtime.py: resolve Deep Nested Complexity in _match_dict_status and _match_list_status via _find_named_container
- config.py: eliminate Bumpy Road Ahead in _format_container_probes and _format_container_env_refs using flat comprehensions
- test_specialists.py: resolve Low Cohesion by consolidating into cohesive test suites sharing SpecialistTestBase

…all methods under 60 lines

…o eliminate code duplication

…structured-specialist-agents

feat(agents): Phase 5 - Structured Specialist Agents (Runtime, Config, Resource)

…omated code quality checks

…lity-automation

feat(ci): configure optimized husky and lint-staged git hooks for automated code quality checks

…ation

- Implement AgentOrchestrator in agents/orchestration/orchestrator.py
- Enforce concurrency limiting using asyncio.Semaphore (default: 2)
- Execute synchronous agent investigations asynchronously via loop.run_in_executor
- Guard each specialist agent with asyncio.wait_for timeout enforcement (default: 25.0s)
- Guarantee complete failure isolation across timeouts and unhandled agent exceptions
- Add comprehensive unit and integration tests in agents/tests/test_orchestrator.py

…ly with CodeScene parameter limits

…parallel-agent-orchestrator

feat(agents): Phase 6 - Parallel Agent Orchestrator with Failure Isolation

- Implement ProgressiveInvestigationLoop in agents/orchestration/tool_loop.py
- Support multi-turn progressive reasoning (ReAct pattern) up to max_turns = 3
- Handle both Ollama native function calling and structured JSON tool actions
- Automatically record all dynamic tool executions into the EvidenceStore
- Enforce hard stop at max_turns and invoke forced synthesis with max depth instructions
- Add unit tests in agents/tests/test_tool_loop.py validating multi-turn tool calling, error feedback, and loop termination
- Add evaluate_next_step and force_synthesis methods to BaseSpecialistAgent

…Scene compliance

…Scene

…phase19-secret-redaction-audit-logging

[KDM-TEST-FEAT-377] feat(observability): Phase 19 - System Observability, Secret Redaction & Audit Logging

…ixtures

… CodeScene 10.0

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

…-multi-agent-test-suite-e2e-fixtures

test(agents): Phase 20 - Comprehensive Multi-Agent Test Suite & E2E Fixtures

- Implement head/tail log clipping in agents/tools/kubernetes.py (clip to 20 head, 80 tail if > 100 lines)
- Configure Ollama chat options with num_ctx=4096, temperature=0.1, and num_predict=512 across BaseSpecialistAgent and specialists
- Minify specialist system prompts by > 50% without loss of diagnostic accuracy or domain boundaries
- Add comprehensive verification and latency benchmark test suite in agents/tests/test_perf_budget.py
- Fixes KDM-cli#292

…ig prompt, and benchmarks

…-concurrency-latency-resource-tuning

perf(agents): Phase 21 - Concurrency, Latency & Resource Tuning

Implements Phase 22 dynamic per-role model strategy and configuration for the Multi-Agent SRE Architecture:

- Created config/agents.yaml supporting dynamic model mapping per role (runtime, config, resource, synthesizer, default)
- Implemented resolve_model(role, config, cli_override) in agents/council.py adhering to priority hierarchy: CLI --model > agents.yaml role mapping > agents.yaml default > fallback 'llama3.1'
- Supported global CLI override via --model in kdm analyze and bridged parameters through python-bridge.ts
- Retained backward compatibility with legacy resolve_model(client, requested_model) invocations
- Created comprehensive test suite in agents/tests/test_model_config.py verifying YAML loading, priority hierarchy, edge cases, and AgentCouncil integration
- Added TypeScript CLI option tests in src/__tests__/analyze-command.test.ts

…odeScene

…-dynamic-multi-model-strategy-config

[KDM-TEST-FEAT-380] feat(agents): Phase 22 - Dynamic Multi-Model Strategy & Configuration

- Register --multi-agent CLI flag in src/commands/analyze.ts
- Add multi_agent boolean configuration option to schema and store in src/config
- Support 'kdm config set multi_agent true/false' via boolean value parser in src/commands/config.ts
- Create consolidated src/config/config.ts re-exporting schema, store, and migration
- Route execution in src/analysis/analysis.ts to runAgentCouncilProcess when multi-agent is enabled via flag, config, or env var
- Emit deprecation notice advising that legacy single-prompt mode will be deprecated in v4.1.0
- Add comprehensive unit test suite in src/__tests__/cli-flags.test.ts and CLI option test in src/__tests__/analyze-command.test.ts
- Fixes KDM-cli#294

…ll test suite concurrency

…nctions for CodeScene code health

…tests

…ature-flags-rollout-legacy-fallback

[KDM-TEST-FEAT-381] feat(cli): Phase 23 - Feature Flags, Rollout Modes & Legacy Fallback

… Validation

… CodeScene code health

…4-final-system-architecture-integration-e2e

[KDM-TEST-FEAT-382] feat(release): Phase 24 - Final System Architecture Integration & E2E Validation